12.3 KB · ⇊ 8 parts · ⤓ Micron 12.3 KB · ⤓ HTML 35.5 KB

top

A neural processing unit (NPU), also known as AI accelerator or deep learning processor, is a class of specialized hardware acceleratorcite-ref-1[1] or computer systemcite-ref-2[2]cite-ref-3[3] designed to accelerate artificial intelligence (AI) and machine learning applications, including artificial neural networks and computer vision.

• Use

• Consumer devices

• Datacenters

• Programming

• Notes

• References

• External links

Their purpose is either to efficiently execute already trained AI models (inference) or to train AI models. Their applications include algorithms for robotics, Internet of things, and data-intensive or sensor-driven tasks.cite-ref-4[4] They are often manycore or spatial designs and focus on low-precision arithmetic, novel dataflow architectures, or in-memory computing capability. As of 2024, a typical datacenter-grade AI integrated circuit chip, the H100 GPU, contains tens of billions of MOSFETs.cite-ref-5[5]

AI accelerators are used in mobile devices such as Apple iPhones, AMD AI enginescite-ref-6[6] in Versal and NPUs, Huawei, and Google Pixel smartphones,cite-ref-7[7] and seen in many Apple silicon, Qualcomm, Samsung, and Google Tensor smartphone processors.cite-ref-8[8]

It is more recently (circa 2022) added to computer processors from Intel,cite-ref-9[9] AMD,cite-ref-10[10] and Apple silicon.cite-ref-11[11] All models of Intel Meteor Lake processors have a built-in versatile processor unit (VPU) for accelerating inference for computer vision and deep learning.cite-ref-12[12]

On consumer devices, the NPU is intended to be small, power-efficient, but reasonably fast when used to run small models. To do this they are designed to support low-bitwidth operations using data types such as INT4, INT8, FP8, and FP16. A common metric is trillions of operations per second (TOPS), though this metric alone does not quantify which kind of operations are being done.cite-ref-13[13]

Accelerators are used in cloud computing servers, including tensor processing units (TPU) in Google Cloud Platformcite-ref-14[14] and Trainium and Inferentia chips in Amazon Web Services.cite-ref-15[15] Many vendor-specific terms exist for devices in this category, and it is an emerging technology without a dominant design.

Graphics processing units designed by companies such as Nvidia and AMD often include AI-specific hardware, and are commonly used as AI accelerators, both for training and inference.cite-ref-16[16]

Mobile NPU vendors typically provide their own application programming interface such as the Snapdragon Neural Processing Engine. An operating system or a higher-level library may provide a more generic interface such as TensorFlow Lite with LiteRT Next (Android) or CoreML (iOS, macOS).

Consumer CPU-integrated NPUs are accessible through vendor-specific APIs. AMD (Ryzen AI), Intel (OpenVINO), Apple silicon (CoreML)cite-ref-17[a] each have their own APIs, which can be built upon by a higher-level library.

GPUs generally use existing GPGPU pipelines such as CUDA and OpenCL adapted for lower precisions. Custom-built systems such as the Google TPU use private interfaces.

cite-note-17a. ↑ MLX builds atop the CPU and GPU parts, not the Apple Neural Engine (ANE) part of Apple Silicon chips. The relatively good performance is due to the use of a large, fast unified memory design.

cite-note-11. ↑ "Intel unveils Movidius Compute Stick USB AI Accelerator". July 21, 2017. Archived from the original on August 11, 2017. Retrieved August 11, 2017.

cite-note-22. ↑ "Inspurs unveils GX4 AI Accelerator". June 21, 2017.

cite-note-33. ↑ citerefwiggers2019Wiggers, Kyle (November 6, 2019) [2019], Neural Magic raises $15 million to boost AI inferencing speed on off-the-shelf processors, archived from the original on March 6, 2020, retrieved March 14, 2020

cite-note-44. ↑ "Google Designing AI Processors". May 18, 2016. Google using its own AI accelerators.

cite-note-55. ↑ citerefmoss2022Moss, Sebastian (March 23, 2022). "Nvidia reveals new Hopper H100 GPU, with 80 billion transistors". Data Center Dynamics. Retrieved January 30, 2024.

cite-note-66. ↑ citerefbrown2023Brown, Nick (February 12, 2023). "Exploring the Versal AI Engines for Accelerating Stencil-based Atmospheric Advection Simulation". Proceedings of the 2023 ACM/SIGDA International Symposium on Field Programmable Gate Arrays. FPGA '23. New York, NY, USA: Association for Computing Machinery. pp. 91–97. arXiv:2301.13016. doi:10.1145/3543622.3573047. ISBN 978-1-4503-9417-8.

cite-note-77. ↑ "HUAWEI Reveals the Future of Mobile AI at IFA".

cite-note-88. ↑ "Snapdragon 8 Gen 3 mobile platform" (PDF). Archived from the original (PDF) on October 25, 2023.

cite-note-99. ↑ "Intel's Lunar Lake Processors Arriving Q3 2024". Intel. May 20, 2024.

cite-note-1010. ↑ "AMD XDNA Architecture".

cite-note-1111. ↑ "Deploying Transformers on the Apple Neural Engine". Apple Machine Learning Research. Retrieved August 24, 2023.

cite-note-1212. ↑ "Intel to Bring a 'VPU' Processor Unit to 14th Gen Meteor Lake Chips". PCMAG. August 2022.

cite-note-1313. ↑ "A guide to AI TOPS and NPU performance metrics".

cite-note-1414. ↑ citerefjouppiyoungpatilpatterson2017Jouppi, Norman P.; et al. (June 24, 2017). "In-Datacenter Performance Analysis of a Tensor Processing Unit". ACM SIGARCH Computer Architecture News. 45 (2): 1–12. arXiv:1704.04760. doi:10.1145/3140659.3080246.

cite-note-1515. ↑ "How silicon innovation became the 'secret sauce' behind AWS's success". Amazon Science. July 27, 2022. Retrieved July 19, 2024.

cite-note-1616. ↑ citerefpatelnishballxie2023Patel, Dylan; Nishball, Daniel; Xie, Myron (November 9, 2023). "Nvidia's New China AI Chips Circumvent US Restrictions". SemiAnalysis. Retrieved February 7, 2024.

• Nvidia Puts The Accelerator To The Metal With Pascal, The Next Platform

• Eyeriss Project, MIT